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ABSTRACT 

To facilitate the secondary analysis of citizenship 
and social studies data coll^-G^JTed by thej National Assessment for 
Educat i on^al Progress, this project constructed a set of three 
consolidated data archives and provided a simplified documentation 
system for users. The three consolidated files produced were (1) the 
1971-72 Social Studies Asses^itent, (2) the 1975-76 Citizenship and 
Social Studies Assessments, and (3) two cohort files for 
9-13-year-olds and for 13-17-^yjBar-olds . The project also explored the 
feasibility of constructing in^s^g^ and scales that «fWould bridge 
across packages and age groups, but. the distribution of the items was 
too thin to allow the^ use of a factor artalysis methodology for the 
identification of comparable items. (Author) 
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ABSTRACT 

To facilitate the secondary analysis of citizenship and social stiidles 
data collected by the National Assessment for Educational Progress, 
this project' constructed a set of three consolidated data archives and 
provided a simplified documentation system for users. The three con- 
solidated files produced were (1) the 1971-72 Social Studies Assess- 
ment, (2) the 1975-76 Citizenship and Social Studies Assessment's, and 
(3) two cohort files for 9-13 year-olds and for 13-17 year-olds. The 
project also explored the feasibility of constructing indices and 
scales that would bridge . across packages and age groups, but the dis- 
tribution of the items was too thin to allow the u3e of a factor anal- 
ysis methodology for the identification of comparable items. 
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^ / ' BACKGROUND OF THE PROJECT 

W 

^^-^ ^ The original design of the sampling and data collection proce- - 

dures for the National Assessment stressed the' concept of repeated 
^ 'measures over time, with each set of measures df'awn from a larger pool 

of items. This design reflected the assessment mission of NAEP. The^ 
result was a dat^ collect5iOI^ pattern that utilized a large number of 
packages of^individU|Ll test items for each of tftree age groups. Al- 
though most of the items are used for most age groups and in all 
years, the distribution of items witihjfn packages follows no pattern 
and thus it has be^n most difficult foi^^ secondary . analysts who use 
single packages or a combination of two or three packages to find the 
same items for cross- tabulation or analytic purposes. 

Although the NAEP has made major improvem^its in the format and * 
availability of the data sets, the basic format is still the test ^ 
^ package. For a secondary .analyst interested in the social studies or 

citlze.iship assessments, this w^uld mean looking at approxii^^eTy 25 
packages for the Year 3 data and about 20 packages for the Y^sr 7 
data. The codebooks alone create a stack of material more than eight 
feet in height. Most secondary analysts are unwilling to take the 
time to sort through these numerous codebooks and to identify those ^ . 

packages of particular interest. Even if a secondary analyst were 
willing to make the effort to search the individual package codebooks 
and find a set of comparable items, many of these potential analysts ^ 

do not have the necessary programming skills (or access to the re- 

# 

sources to hire those skills) to make the necessary file mergers and 
to cprrstruct an consolidated analysis file. 
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As a result fcf these factors, we have observed a relatively low 
rate of use "of the very Valuable NAEP data sets for secondary analy- 
sis. The, purposes of this proposal were (1 ) to investigate the possi- 
bill-ty' of using-)i^ Statistical Package for the Social Sciences (SPSS) 
to construct consolidated files that wo^ld provide a common index of 
variables and (?) to examine the potential for constructing comparable 
Indices across agdl and year groups. The achievemeivji of these two ob- 
jectives shoiad facilitate secondary analysi^ and encourage greater 
use of the NAEP data. 



i WORK COMPLETED 



1^ 



.During the period of the contract, a successful file merger pro- 
cedure was developed' and three major consolidated^Jiles were complet- 
ed. .A series of attempts were made to construct comparable indices 
across age and year groups, but only limited success was obtained. It 
may be useful to describe each set of efforts in more detail and to 
assess- the utility of each outcome. 

The major achievement of the project was the identification of a 
m^l^hod for the- fconsc^idation of 111 of the packages from a given test 
year into a single SPSS save file, with a oommon codebook and variable 
index. The work was done on the Year 3 and Year 7 asse^ments for 
citizenship and social studies and the feasibility of the process was 
demonstrated. The system was presented to the Second Workshop on the 
Secondary Analysis of NAEP Data, held at Northern Illinois University 
in November, X'98l, and was favorably received by a wide range of 
users. y 

• • < 
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'"'^Aa a result of the file construction efforts, three major user 



files and codebooks wer? created: 

This tape file contains all of the individual 
packages for Year 3 from all three age groups, 
A common codebook displays the location of each 
variable and a user may selects items that are 
common across age groups by specifying the 
packages (or subfiles, in SPSS terms) containing 
those items, A single two-inch codebook 
replaces the almost five feet of individual 
package codebooks that would have been 
^ necessary otherwise, 

(Z) ifi^LH 1 C j-tizensh ip/^'^QCial Studies, 

This tapo file contains all of the individual 
packages for Year 7 from all three age groups. As 
above, a common codebook displ^s the location of 
variables across packages and age groups, 

Q) Year '-^-Year 1 Cohort file. 

This tape file includes all of the comparable 
Items from two cohorts that were included in the 
Year 3 and Year 7 assessments^ One file includes 
the 9-year-olds from Year 3 and the 13-y^ar-olds 
from Year 7, which represent two independent 
samples of the same age cohort, four years apart, 
A second file includes the 13-year-old3 from Yegr 
3 and the 17-year-olds from Year 7i which form a 
second set of' two independent samples of a single 
age cohort four years apart. These two files 
provide easy user access to two very important 
data sets that were not previously available 
for secondary ^analysi^ without enormou^s 
programming and documentation effort. 

The data processing for^ the tape files has been completed and the co- 
debooks are included with this report. In the appendices to the code- 
books, the full text of all released items is provided and the short 
^ext of secure items. We have also prepared comparable pages that in- 
clude the full text of the secure/ items (stamped to indicate the re- 

stricted nature of the item) and we would recommend that these addi- 

f 

tional pages be made available to interested scholars by NAEP upon the 
signing of a non-disclosure statement by the individual researcher. 



We are convinced that the quality of the analysis will be improved by 
some form of access to the full text of all items. 

It should be noted that the construction of these files involved 
both extensive programming efforts, which were preformed mainly by 
Janet McConeghy, but also extensive work in "compiling old records from 
the NAEP.^taff for documentation purposes. Both- David Wright and Su- 
san, Oldefendt provided extraordinary assistance in searching the NAEP 
file^ for rec^ds of coding decisions and .variable definitions that 
were invaluable to the file construction process. - 

The second phase of the work involved the effort to construct 
comparable indices across age and year grCups that would provide addi- 
tipnar'tools for the s'^condary analyst. In this effort, our success 
was limited. 

. Working from the pFan outlined in the proposal, gamma matrices 
were con- structed for sets of items from all packages in each age 
group and these matrices were subjected to a factor analysis. It was 
hoped that there would emerge some comparable factors that could be 
compared analytically, even Chough tl^e identical items were not in- 
eluded. This approach assumed that the distribution of items withir^^ 
packages might have teen done purposefully, with a given number of 
items coming from each of the major objectives around khlch the test 
Items was constructed. That assumption turned out'not to be true and 
there is no pattern of item 'assignment within the citizenship/social 
studies packages for either Year 3 or Year 7^. 

In the year 7 dat?i set, for example, we systematically construct- 
ed gamma matripes for each of the "^packages and factor analyzed each 



matrix. lli gneral, one major fac^r emerged for each package^ that ac- 
counted for about Id per .cent of the common varianc^ but only about 
30 per cent of the total variance. It appeared that this factor rep- 
resented a general knowledge or skill , level in citizenship and social 
studies. Although several additional factors usually emerged with ei- 
gen values greater than 1.0, they did nothave an interpretable char- 
acter and generally accounted for only trivial aihounts of variance in 
the matrix. In no case did the pattern of factors resemt^o the objec- 
tive structure, which had been our orginal hope and which mi-ght have 
^provided the foundation for the construction of comparable indices. 

It now appears to us that the relative small number of items in 
each NAEP package and the large number 'of objectives from wnicix they 
are drawn preclude the development of comparable indices across pack- 
ages and age groups that are base,d on more than two or three items. 
If at some future date, the NAEP should reconsider its present sam- 
pling design and package format, some consideration should be given to 
increasing the size of the test packages and to providing for a more 
systematic replication of items across packages and age groups and 
years. 

Since the proposed approach did not work, an alternative was ex- 
plored and promises some success. In studying the distribution of 
items across package^ and years, it appeared that about two- thirds of 
the items do appear ±\ all three age groups,, although not -in the same 
packages in every case. Using attitudes toward law and rules as a 
test case, we found that it was possible to find sets of seven items 
that were comparable across packages in all age groups in Year 7 and 



that this analytic subset constituted a pocfl. of just over 12,000 cas- 
es. This is a very useful stucjy population size, and we found it pos- 



of statistical significance and a good cell distribution. While the 
success of this approach will vary some by the topic that the secon- > 
d^ry analyst investigates, it appears from our work with the codebooks 
that there are a wide range of topics in which the identical items 
will appear in enough individual packages to pro^^ide a case pool in 
the 12,000 range. . 



A full set of materials have been made available to NAEP with 
this report for distribution to the academic and secondary analysis 
communities. On the basis of the first reactions of those social sci- 
ence educators whoN^a«^e seen the system, it appears. that there will be 
a strong demand for these files. Whi^e the final distribution strat- 
egy is ythe responsibility of the NAEP, we would make the fbllowing 
recommendations: 

(1) A set of codebooks and tapes should be depojsited with 
the Inter-University Qonsortiuic for Political and 
Social Research (ICPSR), located at th^ University 

of Michigan. This is the major repository for second- 
ary analysis data sets in the United States and will 
assure wider utilization of. the NAEP Data. 

(2) That NAEP conducts or arranges for introductory user 
orientation sessions at annual meetings like the 



Ameriacan Educational Research Association (AERA) and 
similar educational and social science organizations. 

(3) That NAEP sponsors or co-sponsors short-term training 
workshops to orient new users to the consolidated file 
structure and the computing technologies necessary for - 
their utilization. Training in n^er quantitative 
methods appropriate to the NAEP data (log-linear 
models, LISREL, latent-trait analysis) should also be 
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made available. 



